Faster Depth-Adaptive Transformers

Depth-adaptive neural networks can dynamically adjust depths according to the hardness of input words, and thus improve efficiency. The main challenge is how measure such decide required (i.e., layers) conduct. Previous works generally build a halting unit whether computation should continue or stop at each layer. As there no specific supervision depth selection, may be under-optimized inaccurate, which results in suboptimal unstable performance when modeling sentences. In this paper, we get rid estimate advance, yields faster depth-adaptive model. Specifically, two approaches are proposed explicitly words corresponding adaptive depth, namely 1) mutual information (MI) based estimation 2) reconstruction loss estimation. We conduct experiments on text classification task with 24 datasets various sizes domains. Results confirm that our speed up vanilla Transformer (up 7x) while preserving high accuracy. Moreover, efficiency robustness significantly improved compared other approaches.